Distributed content analysis network

ABSTRACT

A distributed content analysis network uses the processing capabilities of customer-premises equipment as subordinate nodes for analyzing multimedia programs. A master node selects a program and identifies subordinate nodes that are available for analysis, which may include both nodes tuned to the program and idle nodes. The master node divides the program into segments for analysis and instructs each subordinate node to analyze a segment. The subordinate nodes then provide analysis results back to the master node, which may build a metadata profile for the program based on the analysis.

BACKGROUND

1. Field of the Disclosure

The present disclosure relates to providing multimedia content and moreparticularly to providing metadata in conjunction with providingmultimedia content.

2. Description of the Related Art

In general, a multimedia content network will include a provider thatdelivers multimedia content to subscribing end users over media such ascable, satellite, or internet protocol television (IPTV). In some cases,providers will provide metadata for programs, which users may see on anelectronic program guide (EPG).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a network diagram disclosing an embodiment of a distributedcontent analysis network;

FIG. 2 is a partial network diagram of a distributed content analysisnetwork, disclosing a plurality of subordinate nodes such as set-topboxes receiving a multimedia content stream;

FIG. 3 is a block diagram of a subordinate node for use in a distributedcontent analysis network;

FIG. 4 is a block diagram of a master node for use in a distributedcontent analysis network;

FIG. 5 is a diagrammatic view of a segmented multimedia content stream;

FIG. 6 is a flow diagram disclosing a process whereby a master nodeassigns analysis tasks to a subordinate node; and

FIG. 7 is a flow diagram disclosing a process whereby a subordinate nodeperforms an analysis task.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Embodiments of a distributed content analysis network use the processingcapabilities of customer-premises equipment (CPE) as subordinate nodesfor analyzing multimedia programs. A master node selects a program andidentifies subordinate nodes that are available to perform analysis,which may include subordinate nodes tuned to the program as well assubordinate nodes that are idle. The master node may divide the programinto segments for analysis and instruct subordinate nodes to analyzedifferent segments. The subordinate nodes may then provide analysisresults back to the master node, which may build a metadata profile forthe program based on the analysis.

In a particular embodiment, a distributed content analysis networkenables a content provider to harness the processing power of CPE, suchas a set-top box (STB), and use the CPE as a subordinate node forprogram analysis. The STB may have some appreciable processing power,which may be otherwise left idle. But by instructing the STB to performan analysis task, its processing power can be harnessed to generate richmetadata for multimedia content. Conventional multimedia networks mayprovide a limited and static form of metadata including, as an example,metadata that supports an electronic programming guide. In contrast, thedistributed data analysis disclosed herein extends metadata concepts toencompass metadata captured in real time reflecting aspects of a programnot knowable in advance, for example, the demographic composition of acrowd at a football or the names of one or more celebrities at acharitable event.

In conventional networks, a provider showing the program Top Gun mayinclude a minimal description in the EPG, such as, “Top Gun. 1986. PG.110 min. Tom Cruise. Kelly McGillis. Tom Skerritt. A hotshot fighterpilot attends the Navy's top flight school.” Generally, this minimaldescription will have been written and entered manually. But if thesubstantial processing power of the many STBs tuned to Top Gun areharnessed, richer metadata can be gleaned. Facial recognition softwarealong with tracking of time on screen may used to determine that AnthonyEdwards, Val Kilmer, Michael Ironside, and Meg Ryan also playsubstantial roles. The software may also locate smaller roles played byJames Tolkan, Tim Robbins, Adrian Pasdar, Duke Stroud, and Linda RaeJurgens. Pattern matching on the credits may be used to determine thatthe movie was produced by Jerry Bruckheimer, the music was scored byHarold Faltermeyer, and the film was directed by Tony Scott. Patternmatching on closed captioning may be useful for locating and markingfamous lines such as “The plaque for the alternates is down in theladies room,” or “The defense department regrets to inform you that yoursons are dead because they were stupid.” Visual analysis may indicatethat Tom Cruise's character flies an F-14, that the instructor pilotsfly F-4s, that the fictional “MiG 28s” are actually F-5s, and that thefilm also features SH-3 and HH-3F rescue helicopters. These data may becorrelated with a central database of additional information to furtherdetermine, for example, that Tom Cruise was 23 and Kelly McGillis was 28when the film was released, or that it was the eleventh film produced byJerry Bruckheimer.

The STB may also record preferably-anonymous metrics based on userinteractions. For example, it may be found that a certain number ofpeople watched the film, that a certain percentage of those recorded iton a digital video recorder (DVR), that DVR users skipped certaincommercials but watched others, and that certain scenes are popularbased on users re-winding and re-viewing those scenes.

In another example, distributed analysis may be used to analyze asporting event or other live program. For example, in a football game,it may be determined in real-time or near-real-time, that a particularplayer played 28 minutes of a game and that he completed three passes.The score may also be tracked in real time and provided to the EPG, sothat a user scanning through the EPG may be able to see the currentstate of the game without actually tuning to it.

The rich metadata discussed above may be exploited for various purposes.For example, an EPG may include “drill-down” capabilities so that userscan get more details about a specific program, including statistical andpersonal details about persons involved or the program itself. A usermay also be able to pause a program and select a particular portion forfurther details. For example, a user with a DVR may pause Top Gun andselect the fictional MiG-28, whereupon he may be informed that theaircraft is an American F-5E Tiger II. And because some multimediacontent systems are connected to the internet, additional results may beavailable by linking in results from services such as Google orWikipedia.

In another example, a football fan who is watching the University ofTexas (UT)-Texas A&M University (A&M) game on Thanksgiving may be ableto open the EPG and discover that 80% of the crowd are UT supporters andthat the celebrities in attendance include Matthew McConaughey, LanceArmstrong, and Vince Young.

Other uses of rich metadata will be made apparent by the presentdisclosure.

A distributed content analysis system will now be described with moreparticular reference to the attached drawings. In the followingdescription, details are set forth by way of example to facilitatediscussion of the disclosed subject matter. It should be apparent to aperson of ordinary skill in the field, however, that the disclosedembodiments are exemplary and not exhaustive of all possibleembodiments. Throughout this disclosure, a hyphenated form of areference numeral refers to a specific instance of an element and theun-hyphenated form of the reference numeral refers to the elementgenerically or collectively. Thus, for example, widget 12-1 refers to aninstance of a widget class, which may be referred to collectively aswidgets 12 and any one of which may be referred to generically as awidget 12.

FIG. 1 is a network diagram of selected aspects of an embodiment of adistributed content analysis network 100. A media source 114, which maybe operated by a multimedia service provider such as a televisionservice, provides a multimedia content stream 122 for use in thenetwork. Multimedia content stream 122 may be provided to a satellitetransmitter 112, which transmits the stream to a satellite 120.Satellite 120 transmits multimedia content stream 122 to one or morecontent access networks 180. Content access network 180 may include asatellite receiver 136 connected to a subordinate node 130, which may beimplemented as an STB or another suitable type of CPE. In standardoperation, subordinate node 130 provides video or multimedia displayoutput 134 to display 150 for viewing and/or listening by user 160. Insome embodiments, subordinate node 130 also interfaces to a broadbandconnection 132, which permits backhaul communication with the serviceprovider via network 140.

A master node 110, which may be a backend server operated by themultimedia service provider, also receives multimedia content from mediasource 114. Master node 110 may be configured to operate a distributedcontent analysis network according to the present disclosure.

Although FIG. 1 discloses an embodiment that uses a satellite network,wired networks such as cable or IPTV networks would also be suitable andwould operate on analogous principles.

FIG. 2 is a diagram of selected aspects of an embodiment of distributedcontent analysis network such as the distributed content analysisnetwork 100 of FIG. 1, illustrating a plurality of subordinate nodes 130such as set-top boxes receiving a multimedia content stream 122. Eachsubordinate node 130 has the ability to tune to or otherwise receive aprogram selected by the user, and in some embodiments, analysis tasksare tied to the selected program. For example, subordinate node 130-1and subordinate node 130-2 may be tuned to Top Gun and subordinate node130-3 may be tuned to the UT-A&M game. Additional subordinate node 130may be included in the network, and each may be tuned to a program. Somesubordinate node, such as subordinate node 130-n, may be idle, meaningthat they are not currently displaying any program to a user. Eachsubordinate node shown may be configured to operate as a subordinatenode in distributed content analysis network 100 (FIG. 1).

Master node 110 may determine that sufficient subordinate nodes 130 aretuned to Top Gun to perform the analysis tasks, but an insufficientnumber of nodes are tuned to the UT-A&M game. In some cases,“sufficient” may mean 1 or more and “insufficient” may mean 0. In otherembodiments, “sufficient” may mean greater than or equal to N where N isthe number of segments. Returning to the Top Gun example, master node110 may assign analysis tasks associated with Top Gun to subordinatenode 130-1 and subordinate node 130-2, and analysis tasks associatedwith the game to subordinate node 130-3. Because there are not enoughnodes tuned to the game, master node 110 may also assign some idlenodes, such as subordinate node 130-n analysis tasks associated with thegame. In the case of idle nodes such as 130-n, it may also be necessaryto provide the program segment to be analyzed.

FIG. 3 is a block diagram of selected aspects of an embodiment of asubordinate node 130 for use in a distributed content analysis network110 (FIG. 1). In the embodiment disclosed, a bus 314 interconnectsvarious system components. Content interface 320 receives multimediacontent stream 122. Network interface 370 communicatively couplessubordinate node 130 to broadband connection 132. In some embodiments,content interface 320 and network interface 370 may be a single physicalinterface (for example, in an IPTV system, multimedia content will beprovided via broadband connection 132). A video adapter 360 is alsoshown, which is configured to provide display output 134 to displaydevice 150. Storage 340 may be a hard disk, solid-state drive, flashmemory, or other type of non-volatile memory device. Memory 312 mayinclude computer executable instructions, i.e., software, and data fromstorage 340, and may be a volatile, low-latency mechanism such as randomaccess memory (RAM). During instruction execution, necessary algorithms,instructions, and data may be temporarily stored in memory 312. Amongother processes, memory 312 may include program analyzer software 330-1and user interface software 350, which may provide a graphical displaythrough which a user may interact with subordinate node 130. Processor310 may execute software instructions from memory 312 or elsewhere toperform processing functions and regulate interactions between othercomponents. As is known in the art, processor 310 may be acentral-processing unit (CPU) or other general-purpose orspecial-purpose hardware performing a similar function.

Although program analyzer software 330-1 may be purely standalonesoftware, it may also be used in conjunction with special purposehardware. For example, program analyzer software 330-1 may operate inconjunction with program analyzer hardware 330-2, which may represent anASIC (application specific integrated circuit) performing a specifictype of pattern matching function, such as text-to-speech, voicerecognition, face recognition, or text matching. This may be moreefficient than a software solution because the particular algorithm willbe implemented in hardware. A planned or random distribution ofsubordinate nodes with different hardware capabilities may be sent tocustomers, so that some will be dedicated to certain tasks. In otherexamples, program analyzer hardware 330-2 may include an FPGA (fieldprogrammable gate array) or other re-configurable device. Programanalyzer hardware 330-2 may be designated for a time as a text-to-speechunit, and program analyzer software 330-1 may be programmed to performtext-to-speech. If there is a greater need later for facial recognition,program analyzer hardware 330-2 may be re-purposed as a facialrecognition chip. Even when dedicated hardware is used, it may benecessary to include certain software instructions in memory 312. Insome embodiments where at least part of a program analyzer isimplemented as program analyzer hardware 330-2, program analyzersoftware 330-1 may be a part of or share resources with processor 310.

FIG. 4 is a block diagram of a master node for use in a distributedcontent analysis network. As with subordinate node 130, master node 110includes a processor 410, a bus 414, a content interface 420, a networkinterface 470, and memory 412. These logical units may be similar to thecorresponding logical units in subordinate node 130, and thedescriptions of those corresponding components above are generallyapplicable to the logical units of master node 110. Master node 110 mayalso include a number of modules dedicated to specific activity.Although these modules are shown resident in memory, as with programanalyzer 330 (FIG. 3) of subordinate node 130, these functions may beimplemented in an appropriate mixture of software and/or dedicatedhardware.

Subordinate activity module 430 may be configured to determine whichsubordinate nodes are idle, which are active, and for active subordinatenodes, which subordinate nodes are tuned to which program. Assignmentmodule 440 determines how many subordinate nodes are available for acertain program. Based on the number of available subordinate nodes,assignment module 440 may subdivide the program into functional,spatial, and/or temporal segments. For example, assignment module 440may designate a 5-minute temporal segment of a program for facialrecognition, or it may designate a certain sector of the screen forfacial recognition, or some combination thereof. In other embodiments, aprogram may be analyzed in its entirety by a single subordinate nodewithout segmentation.

Analysis reception module 450 may receive and process results of theanalysis after subordinate nodes 130 have completed their tasks.Assignment module 440 and analysis reception module 450 may operate in afeedback loop. For example, assignment module 440 may first assign anumber of nodes to perform high-level analysis, to determine thatcertain portions of the program primarily feature people, other portionsprimarily feature vehicles, and some portions feature crowds of people.Master node 110 may use this high-level analysis as a basis forassigning more specific tasks, such as assigning subordinate nodes toperform facial recognition on people in a particular scene. Thisfeedback loop may also be iterative. For example, on a first pass,subordinate nodes 130 may determine that a certain scene primarilyfeatures people. Master node 110 may then divide the scene into spatialsegments, with each segment including one or more faces for analysis,and may assign certain subordinate nodes 130 the task of characterizingthe faces in the scene. Once subordinate nodes 130 have returnedcharacteristic profiles of the faces, master node 110 may search adatabase of celebrities to find detailed profiles. Master node 110 maythen send a subset of celebrities matching the profile to certainsubordinate nodes 130, and instruct the subordinate nodes 130 to performa more detailed analysis. On this iteration, one or more subordinatenodes may match the face to a celebrity with a particular degree ofcertainty.

Based on matches to certain patterns or other data gleaned from theanalysis process, metadata update module 460 may select some or all ofthe data for inclusion in a metadata profile. Metadata update module 460may include algorithms for classifying types of data and compiling thoseinto a profile, which are updated as program metadata 480.

FIG. 5 is a diagrammatic view of a segmented multimedia content stream500. FIG. 5 illustrates that a program may be both temporally andspatially divided. For example, temporal segment 512 may be part of aprogram timeline 510. Temporal segment 512 illustrates a case where a20-minute portion of Top Gun has been designated for analysis. On theother hand, multi-element image 520 includes a large number of faces,which may represent a portion of the crowd attending the UT-A&M game.Spatial segment 522 is designated for analysis. This may be useful, forexample, for estimating the demographic composition or schoolaffiliation of the crowd for statistical analysis, or simply foridentifying celebrities attending the game live.

FIG. 6 is a flow diagram disclosing a process whereby a master nodeassigns analysis tasks to a subordinate node. In block 610, master node110 identifies a program to be analyzed. In block 620, master node 110identifies subordinate nodes that are available for analysis (forexample, subordinate nodes that are either idle or that are tuned to theprogram). In block 630, the program is divided into discrete segments,which may be any combination of spatial, temporal, and/or functionaldivisions. For example, a subordinate node 130 may be instructed toperform facial recognition (a functional division) on a 10-minutesegment (a temporal division) in the upper-left quadrant of the screen(a spatial division). In block 640, analysis tasks are distributed tosubordinate nodes. In block 650, master node 110 receives the results ofthe analysis. In block 660, the master updates the metadata 480 (FIG. 4)associated with the program.

FIG. 7 is a flow diagram disclosing a process whereby a subordinate node130 performs an analysis task. In block 710, subordinate node 130receives tuning instructions from user 160, associating subordinate node130 with a particular program. Alternatively, in block 712 subordinatenode 130 may be idle. In block 720, subordinate node 130 requests theprogram. In block 730, subordinate node 130 receives the program fromthe multimedia service provider. In block 750, subordinate node 130displays the requested program to user 160. While user 160 is viewingthe program, or while subordinate node 130 is idle, subordinate node 130may perform an analysis task. In block 740, subordinate node 130receives an analysis task from master node 110. In block 760,subordinate node 130 isolates the assigned program segment from theprogram. Note that in some cases of temporal segmentation, the streammay not yet be at this point in the program. In that case, subordinatenode 130 will either wait for that portion of the program, or masternode 110 may provide the assigned segment separately so that subordinatenode 130 can begin analysis. In block 770, subordinate node 130 analyzesthe multimedia content stream 122 according to its instructions. Inblock 780, subordinate node 130 delivers the results of its analysistask to master node 110.

To the maximum extent allowed by law, the scope of the presentdisclosure is to be determined by the broadest permissibleinterpretation of the following claims and their equivalents, and shallnot be restricted or limited to the specific embodiments described inthe foregoing detailed description.

1. A master node in a distributed content analysis network, the masternode comprising: a memory device, the memory device programmed withinstructions to: identify a program in a content stream provided to aplurality of subordinate nodes; assign analysis tasks to the subordinatenodes, each analysis task including instructions to analyze a portion ofthe program; receive results of the analysis tasks from the subordinatenodes; and generate a metadata profile of the program based, at least inpart, on the analysis results.
 2. The master node of claim 1 wherein theportion of the program is a spatial segment of a video component of theprogram as displayed on a screen.
 3. The master node of claim 1 whereinthe portion of the program is a temporal segment of the program.
 4. Themaster node of claim 1 wherein the analysis task includes instructionsto perform pattern matching on the content stream.
 5. The master node ofclaim 4 wherein the pattern matching includes performing facialrecognition to search for a match among a set of pre-selected celebrityfaces.
 6. The master node of claim 4 wherein the pattern matchingincludes parsing an audio component of the content stream to performvoice recognition to search for a match among a set of pre-selectedcelebrity voices.
 7. The master node of claim 4 wherein the patternmatching includes performing speech-to-text conversion on a portion ofan audio component of the program to perform pattern matching on spokenphrases.
 8. The master node of claim 4 wherein the pattern matchingincludes parsing a closed-captioning component of the content stream toperform pattern matching on captions.
 9. The master node of claim 1wherein the content stream is a broadcast of a sport contest and theanalysis task includes instructions to characterize events within thecontest.
 10. The master node of claim 1 wherein said assigning comprisesassigning the analysis tasks to a subordinate node currently tuned tothe program.
 11. The master node of claim 1 wherein said assigningcomprises assigning the analysis task to an idle subordinate node. 12.The master node of claim 1 wherein the subordinate nodes comprisecustomer premises equipment.
 13. The master node of claim 12 wherein thecustomer premises equipment comprises a set-top box.
 14. A subordinatenode for use in a distributed content analysis network, the subordinatenode comprising: a processor configured to execute softwareinstructions; a multimedia content interface communicatively coupled tothe processor and configured to receive a multimedia content stream andprovide the multimedia content stream in a format suitable for use bythe processor; a network interface configured to communicatively couplethe processor to a network; a stream analyzer configured to analyze aportion of the content stream; and a storage medium communicativelycoupled to the processor and containing executable software instructionsfor the processor that when executed instruct the processor to: receivea directive from the master node, the directive instructing theprocessor to analyze a portion of the multimedia content stream; isolatethe portion from the content stream; provide the isolated portion to thestream analyzer; direct the stream analyzer to analyze the isolatedportion; receive analysis results from the stream analyzer, wherein theanalysis results indicate a property of the content stream; and providethe analysis results back to the master node.
 15. The subordinate nodeof claim 14 wherein the property of the content stream is selected fromthe group consisting of the presence of a celebrity face, the presenceof a celebrity voice, the presence of a phrase, and the state of asporting event.
 16. The subordinate node of claim 14 wherein the streamanalyzer includes a software subroutine stored on the storage medium.17. The subordinate node of claim 14 wherein the stream analyzerincludes a dedicated hardware component.
 18. The subordinate node ofclaim 14 wherein the subordinate node is configured to display the fullcontent stream to a viewer while the stream analyzer analyzes theisolated portion.
 19. The subordinate node of claim 14 wherein thesubordinate nodes comprise customer premises equipment.
 20. Thesubordinate node of claim 19 wherein the customer premises equipmentcomprises a set-top box.
 21. A method of analyzing a content stream, themethod comprising the steps of: enabling a master node to be placed incommunication with a plurality of subordinate nodes; and enabling themaster node to: identify discrete portions of a multimedia contentstream for analysis; distribute analysis tasks to the subordinate nodes,each analysis task including directives to analyze a discrete portion ofthe multimedia content stream; receive analysis results from thesubordinate nodes, wherein the analysis results indicate a property ofthe content stream; and consolidate the analysis results into a metadataprofile for association with the multimedia content stream.
 22. Themethod of claim 21 further comprising the steps of: enabling thesubordinate nodes to: receive the multimedia content stream; receive thedirectives from the master node; in response to the directives, analyzea discrete portion of the multimedia content stream; and provideanalysis results to the master node.
 23. The method of claim 21 whereinthe discrete portion of the multimedia content stream includes atemporal segment of the multimedia content stream.
 24. The method ofclaim 21 wherein the discrete portion of the multimedia content streamincludes a spatial segment of a video component of the multimediacontent stream.
 25. The method of claim 21 wherein the discrete portionof the multimedia content stream includes an audio component of themultimedia content stream.
 26. The method of claim 21 wherein thediscrete portion of the multimedia content stream includes a closedcaptioning component of the multimedia content stream.
 27. A computerstorage medium in a master node, the storage medium including softwareinstructions that, when executed, instruct the master node to: parse acontent stream into discrete segments; locate a plurality of availablesubordinate nodes; assign analysis tasks to the subordinate nodes basedon the discrete segments; receive results of the analysis tasks from thesubordinate nodes; and consolidate the results from the discretesegments into a unified metadata profile for association with thecontent stream.
 28. The computer storage medium of claim 27 wherein theinstructions to locate a plurality of available subordinate nodesincludes instructions to identify nodes tuned to the content stream andselect the nodes as available subordinate nodes.
 29. The computerstorage medium of claim 27 wherein the instructions to locate aplurality of available subordinate nodes include instructions toidentify nodes not actively displaying content to end users andselecting the nodes as available subordinate nodes.
 30. The computerstorage medium of claim 27 wherein the subordinate nodes comprisecustomer premises equipment.
 31. The computer storage medium of claim 30wherein the customer premises equipment comprises a set-top box.